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ABSTRAC" 

A Study exBttined the question of whether there is a 
connection between prograa characteristics and desirable prograa 
outcoaes fstudent satisfaction, placeaent, and lob perforaance 
ratln<js». A prograa evaluation questionnaire, containing over 300 
Individual Iteas on such prograa characteristics as teacher 
certification and experience, instructional objectives, perforaance 
standards, coaaualty relations, and counseling services, was used to 
collect data for over 11,000 students and 600 teachers froa Maryland 
vocational prograas at the secondary level. In addition, followup 
supervisor ratin9s of job perforaance were available for 
approxliately BOO students. Data froa these sources were coapiled and 
analyzed with respect to the relationship between 36 prograa 
characteristics (in the areas of advisory councils, adainistration, 
public relations, staff, facilities, recruitaent, counseliag, 
curriculum, student organizations, and placeaent services) and the 
three prograa outcoaes. Relationships between characteristics and 
outcoaes were presented where consistent trends were found and 
several speculations were aade concerning these relationships. 
Recoaaetidations called for further research involving sufficient 
saaples of students of each sex and race. (A related coaparisoa of 
four alternative delivery systeas for vocational education is 
available separately through ERIC--see note.! (HN) 
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Background to the Present Studies 

For the past several years, the Maryland State Department of 
Education's Vocational-Technical Divisior. has been developing, with 
the help of the Educational Testing Service, a program evaluation 
questionnaire. The questionnaire is completed by local personnel, 
students, and members of a visiting team. The items on the question- 
naire inquire about teacher certification and experience, instructional 
objectives, performance standards, community relations, counseling 
services, and other program characteristics. A copy of this question- 
naire can be found in Appendix A'. 

Because the questionnaire was designed, to comprehensivf ly cover 
all important program characteristics, it is quite lengtny. It is 
12 pages long and contains over 300 individual items. A prior study 
by the present principal investigator (Johnson, 1980) was conducted to 
help reduce the number of program characteristics down to a central, 
essential set of categories. This was accomplished by mailing a letter 
to 50 State Directors of Vocational Education and to vocational educa- 
tion administrators in the District of Columbia and five U.S. ter- 
ritories, asking them for a list of what they considered to be essential 
indicators of program quality. 

Examination of the returns showed 12, common themes or categories of 
program quality. Briefly, these were: (1) active advisory council and 
craft committees; (2) effective administration of program policies; 
(3) written plan for public relations; (A) certified, qualified staff; 
(5) adequate facilities and equipment; (6) recruitment program with 
equal access; (7) guidance and counseling services; (&) realistic, 
competency-based curricula; (9) cooperative education and supervised 
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work experience; (10) student organizations; (11) placement and follow-up 
services; and (12) program evaluation. 

The first study in this final report describes how the quality indi- 
cator research project was usad to organize and analyze data already 
collected with the ETS program evaluation questionnaire. Questionnaire 
data were available for over 11,000 students and over 600 teachers from 
Maryland vocational programs at the secondary level. ^ The second study 
compares four alternative delivery systems for vocational education — 
apprenticeship, CETA, cooperative education, and industrial training-^and 
examines the relevance of the 12 dimensions of program quality for these 
systems . 



Thanks go to Leo Lezzer for providing this archival data. 



Study I: Relationship between Proposed Vocational Program Quality Indicators, 

Student Satisfaction, Placenent, and Jbb Performance Ratings 
Introduction 

The study of program quality indicators drew a distinction between 
program characteristics and program outcomes . Program characteristics 
are features of the program itself, such ?s the Jreachers, the physical 
plant, the counseling services, the curriculum, and so forth. Program 
oucowes include placement rate, student satisfaction, and employer ratings 
of job performance. The 12 categories of quality indi ate s identified 
by Johnson (1980) are all program characteristics. 

One concern expressed in the above study was whether there was a 
connection between quality indicators (program characteristics) and 
desirable program outcomes. For example, there was nationwide consensus 
that the presence of an active advisory council is an essential quality 
indicator; it r'-mains to be seen, however, whether programs with an active 
advisory council place more students, create high levels of student satis- 
faction, etc. The present study answers that question by comparing 
program characteristics with program outcomes. 
Method 

The items on the Edt'c^tional Testing Service questionnaire were first 
sorted into those describing program characteristics and those dc^scribing 
program outcomes. In teriis of outcomes, the questionnaire yielded items 
related to four measures cf student satisfaction (with instruction, facil- 
ities, counseling services^ and student organizations), placement rate for 

Jes, and placement rate for females. In addition supervisor ratings of 
job performance were available for approximately 800 students who had been 
followed up. These ratings included judgements of job knowledge, quickncBs 
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in learning job skills, work attitude, ability to work wi^h others, and 
overall work performance. 

The next task was to sort the items dealing with program character- 
istics into categories defined by the 12 dimensions of program quality. 
The content of the items on the questionnaire were such that 10 of the 
12 dimensions of quality were represented. Each dimension of quality 
contained between 1 and 6 subcomponents; this meant that a total of 
36 program characteristics — each a proposed quality indicator from the 
Johnson (1980) study—could be scored from the ETS questionnaire. 

The specific assignment of ETS questionnaire items to the 36 program 
characteristic scales and the program outcome scales is described in 
detail in Appendix A. Essentially, the following proceJure was used. 
First- nly programs for which employer job perforaance ratings were 
available were selected for analysis. Job performance ratings were 
available for 56 programs. Within each program, the number of people 
who rated that program's characteristics, and the number of program 
graduates rated by employers varied considerably. Therefore, average 
ratings cf program characteristics and program outcomes were ccfmputed. 

Pearson correlation coefficients were computed between the 36 
program characteristics (quality indicator) scores assigned to each 
program and the il outcome scores. The result of this analysis is 
a matrix of 396 coefficients, presented in Table 1. Because the pro- 
gram constitutes the unit of analysis, the sample size is N = 56. 
A discussion of tliis table follows. 
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Table 1: Relation between Program Characteristics and Outcomes 

Employer Ratings 



Placer ent 



Advisory Council 
Help Programs 
Composition 
Meetings 
Jiev^un i ca t i on 
Guidelines 

Administration 
Written Policies 
Eliminate Biases 
Support Staff 

Public Relations 

Written Material 

Media Use 
* • 

Staff 

Certification 
Work Experience 
Inservice 

Professional Orgn. 
Student O^gn. 

Facilities 

Replicates Work Sit. 
Equipment Inventory 
Safety 

Accessibility 
Layout 

Recruitment 
Outreach 

Minority Enrollment 
Consider Empl. Oppt, 

Counseling 



Cur 


Fac 


Cns 


Org 


Knw 




Att 


Inp 


Ovl Female Male 


37** 


25 


13 


-n 


18 


08 


-08 


-02 


12 


13 


12 


20 


09 


03 


-07 


04 


-06 


-23 


-23 


-24 


27* 


15 


17 


13 


13 


-40** 


19 


09 


-12 


-07 


19 


47** 


51** 


15 


11 


21 


-50** 


04 


-05 


-21 


-32**-08 


41** 


58** 


22 


16 


21 


-23 


11 


06 


-16 


-1 3 


08 


18 


14 


18 


02 


04 


02 


-05 


05 


06 


1 7 


01 


03 


16 


-17 


-03 


-14 


-02 


-11 


-13 


-08 


13 


-18 


01 


26* 


-15 


-29* 


-20 


29* 


02 


02 


09 




-22 


03 


-14 


11 


03 


26* 


_41** 


-05 


11 


04 


-01 


-06 


09 


04 


-10 


-17 


22 


-18 

» 


-02 


21 


-05 


-08 


-06 


36** 


06 


-07 


-07 


16 


28* 


-02 


02 


04 




16 


-04 


-04 


43** 30** 02 


-02 


08 


17 


-09 


-02 


-07 


-26* 


15 


-02 


-04 


10 


-22 


17 


18 


07 


12 


24 


-13 


08 


15 


09 


26* 


49** 


05 


-09 


-12 


-03 


-18 


-08 


09 


-22 


-25 


-19 


12 


-22 


-11 


-13 


-13 


-24 


04 
<» 


08 


11 


16 


-09 


-04 


-38**- 34**- 20 


-18 


-13 


-15 


12 


05 


04 


09 


-28* 


18 


01 


-04 


-10 


-08 


-15 


07 


2lt 


31* 


09 


-26* 


-03 


-OS 


-27* 


-22 


_34** 


-33* 


-09 


03 


-08 


-16 


-22 


32* 


29* 


15 


24 


16 


-19 


25 


-04 


05 


15 


_54** 


11 


-09 


-09 


-02 


-11 


32* 


2j 


14 


-03 


04 


-09 


03 


01 


-02 


-11 


-04 


08 


19 


-09 


-19 


-05 


-23 


12 


15 


04 


00 


02 


15 


06 


-02 


04 


22 


-12 


-12 


05 


16 


27 


-01 


09 


00 



Career Planning 


-03 


-01 


04 


-07 


12 


-02 


29* 


16 


24 


Clear Roles 


06 


-01 


03 


10 


00 


07 


17 


13 


10 


Availability 


19 


25 


15 


08 


-24 


_37**. 


-26* 


-25 


-29* 


Job Information 


22 


15 


20 


06 


-03 


-03 


21 


25 


15 


Number of Counselors 


-02 


-09 


13 


-15 


23 


34** 


31* 


23 


00 


Needs Assessment 


-19 


-09 


04 


16 


-16 


-21 • 


-06 


-04 


-07 



49** 10 
-10 -05 
-13 16 
37** 05 
42**-02 
14 45** 
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Table 1, con*t 



Program Characteristic Student Satisfaction Employer Ratings Placement 





Cwr Fac 


Cns 


Org 


Rnw 




Att 




Ovl 


Female Male 


Curriculum 




















Relevancy 


-01 -11 


-05 


-04 


01 


16 


03 


11 


11 


-09 -04 


Task Analysis 


39** 27* 


20 


-13 


-14 


-05 


-21 


-13 


-16 


-21 -05 


Varied Methpds 


20 OJ 


23 


15 


05 


04 


01 


00 


,10 


-13 -15 


Outside Resources 


12 11 


04 


-12 


-02 


01 


05 


04 


13 


-05 02 


Student Organizations 


-18 -16 


-22 


13 


-10 


-02 


-09 


06 


-22 


1.1 11 


Placemtent S2rvices 


33* 03 


12 


57** 


-03 


18 


10 


03 


-05 


43** -02 



*p less than .05; **p less than .01 

Note. Decimal points are omitted from all correlation coefficients. Abbreviations 
for student satisfaction are as follows: Cur = curriculum; Fac facilities; 
Cns - counseling services; Org = student organizations. Abbreviations for 
employer ratings are as follows: Knw ~ job knowledge; Qck = quickness in learning 
new job skills; Att - work attitude; Inp " Interpersonal skills; Ovl = overall 
rating. A complete description of the scoring scheme for these variables is 
presented in Appendix A. 
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Results and Discussion 

In a table of 396 Pearson corr'elation coefficients, about 20 would be 
expected to reach statistical significance at the .05 level, due to chance 
alone. To single out only statistically significant correlations for dis- 
cussion would therefore be unwise. A more profitable strategy would be 
to use some interpretive judgment and look for consistent patterns of 
relationships between program characteristics and program outcCwQes. -That 
means that if all of the subcomponents of a program characteristic were 
correlated in the same direction (either positively or negatively) with 
all measures of student satisfaction, this suggests that a real relation- 
ship exists, even if a majority of the coefficients do nor i -^ach the .05 
level of statistical significance. 

For example (and this is purely hypothetical), the three subcomponents 
of Administration might correlate about -.21 to -.24 with the measures of 
student satisfaction and about .19 to .25 with the employer job performance 
ratings and placement rate. Although none of these correlations significant 
in a strict statistical sense, the pattern of negative correlations implies 
that programs with a highly-rated Administration have lower levels of 
student satisfaction, but good employer ratings and a high placement rate. 

Looking at the data this way, relationships between characteristics 
and outcomes will be presented where consistent trends are found. The 
findings are presented for each category of quality indicators separately. 
No attempt to explain these relationships will be made in this section of 
the report; that is reserved for the Speculation section. The present 
section merely describes the findings. 

St, 5 of the 10 categories of program characteristics showed no 
consistent relationships with any of the outcomes. These were: Administration, 
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Public Relations, Facilities*, Recruitment, and Student Organizations. 
This is not to say that these features are not important; rather, as 
they were analyzed here, they seemed to have little effect on outcomes. 

All five subcomponent scores for the Advisory Council / Craft 
Committee dimension (Helps Programs, Balanced Composition, Regular 
Meetings, Connnunication, Written Guidelines) showed consistent re- 
lationships with three of the four measures of student satisfactr'.on 
and with both male and female placement rate. It wculd appear then, 
that in terms of the satisfaction and. placement outcomes, the existence 
of an active adyisory council is Ik ^ad an indicator of program quality. 

Two of the five subcoiSponent scoreg.for Staff appeared to be re- 
lated to outcomes. First, ^ in programs where the staff had mote work 
experience and were involved in professional organizations, student 
satisfaction was higher. Second, in programs whose teachers attended 
inservice meetings frequently, employers tended to rate the students' 
work performance more highly. Thus, professional involvement on the 
part of the teaching staff seer to have a positive effect on the 
students ' education . 

Counseling* services were related to outcomes in the following ways. 
The presence of a well-organized system of testing, planning, and record- 
keeping war; associated with a nigh placement rate (especially for females) 
and with employer job ratings (particularly of work attitude and working 
well with others). Next, having counselor roles and responsibilities 
clearly definei^ was associated wita good job performance ratings, again 
mostly work atttttrde and working well with others. 

The subcomponents of the counseling dlimension dealing with the direct 
relationship bet^^en counselors and students showed several expected, but 



one unexpected finding. First, an one might predict, the amount of 
information counselors gave about job availab'*'' Ity was related to 
student satisfaction, job performance rat.ings, and placement for 
females. Next, in nrograms with a greater number of r junselors, job 
performance ratings were higher, as was the placement rate for females. 
Then, in an unexpected finding, counseling availability (how often 
counseling was offered) was positively associated with student satis- 
faction, but negatively related to all job performance ratings pro- 
vided by the employers. (Possible recsons for this finding are pre- 
sented in the Speculations section.) Finally, in programs that 
stressed assessment of counsel -"ng needs, placement rate was higher for 
both males^ and females. 

Turning to the curriculum, four of the five components were posi- 
tively related to rtudent satisfaction. In order, these were: using 
task analysis as a basis for instruction, using varied instructional 
techniques, using outside resources, and having written objectives 
available for each course. 

Finally, programs with higher ratin;^s of their placement serv.-es 
had a higher level of student satisfaction, and a higher placement 
rate — but for females only. 
Speculations 

Overall, it appears that the Advisory Council / Craft Committee, 
Staff, Counseling, Curiiculum, and Placement Services are the program 
characteristics that have the greatest impact on program outcomes. The 
relationships between the other dimensions of program characteristics 
and program outcomes were weak, inconsistent, or nonexistent. 

if association between a program characteristic and a desirable 
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program outcome is a requirement for calling a program characteristic 
a genuine quality Indicator , then only five of 12 proposed dimensions 
of quality pass the test, and not all of the subcomponents within these 
five dimensions meet this requirement. Intuitively, one might think 
that if a majority of state directors for vocational education agreed 
that a program characteristic is an indicator of quality, that character- 
istic should be related to desirable outcomes. One possibility why not 
all of the program characteristics Judged important by the state directors 
IS quality indicators were associated with outcomes is that these program 
characteristics are important for legal reasons (e.g., policy against sex 
and race bias) or political reasons (e.g., public relations), but do not 
have a direct impact on variables like student satisfaction or employer 
ratings of performance. The limitations of the available data were such 

that only a narrow range of outcomes could be examined. The relationship 

2 

between program characteristics and other program outcomes could be 
addressed by future resear^^'i. 

Another r^sason why t »nships were not found between outcomes and 

all of the program characteristics could be the technic il limitations of 
the study. These are discussed in detail in the following section, 
Limitatio-^s and Suggestions for Future Research. Speculation about the 



Possible outcomes for study are: program completion, demonstrated skill 
proficiency. Job stability, student demand, student Job satisfaction, wages 
minority enrollment, rate of job advancement, cost/benefit ratios, return 
rate, active employer recruitment, program reputation in community, rpferal 
endorsement by professional groups, and community support. Negative out- 
comes wouW include physical injuries, failures, withdrawals, absenteeism, 
tardiness, behavior problems, poor social adjustment, and teacher turn-over 
rate* These outcomes were gathered by Johnson (1980) but not listed in 
that report. 
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relationships that were found are now presented. 

A full 80 percent of the state directors who contributed quality 
indicators in the Johnson (1980) study agreed that Advisory Councils 
and Craft Committees are an integral part of a vocational education 
program. In that report it was noted that some ambigui^y existed on 
precisely what an advisory council was what its functions were. The 
term could refer to a state committee serving all of the programs in 
the state, a local committee serving a school* s overall program, or 
specific committees serving each topical program area with the school's 
overall program. It is likely that in the present study, it is the 
local, specific advisory personnel that are having the greatest impact 
on outcomes. These local per£,onnel have intimate knowledge of the 
working conditions, practices, and employment opportunities in their 
fields. It is not surprising that programs that use this valuable 
information have higher levels of student satisfaction and good place- 
ment rates. 

Neither was it surprising to find positive outcomes in programs 
whose tea-hers had more work experience, were involved in professional 
organizations, and attended inservice meetings frequently. First, 
teachers who are professionally active acquire job knowledge that is 
more extensive, accurate, and up-to-date than teachers who simply teach 
from old textbooks. This certainly benefits students. Just as important, 
however, is that professional involvement indicates enthusiasm and a 
positive attitude about teaching; this is reflected in student satisfaction. 

Examination of the counseling dimension showed that overall organ- 
ization of counseling services was associated with a higher placement 
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rate and favorable ratings by employers for job performance, especially 
along the social-interpersonal dimensions (work attitude and ability to 
work with others). This makes sense in light of a study by Holland, 
Gottfredson, and Power (1980). Holland et al. suggest that two important 
functions of career counselors are dealing with general maladjustment and 
lack of information about jobs. Effective counseling should therefore 
be associated with (a) good work attitudes and interpersonal effectiveness, 
and (b) knowing enough about job availability to secure employment. This 
is precisely what the present study found. 

That the availability cf counseling is negatively associated with 
employer ratings of job performance is puzzling. A possible explanation 
is that in programs in which counseling is always readily available, 
students develop a dependent relationship with their counselors. Upon 
graduation they move into a work environment that requires a high degree 
of independence and self-reliance. In this situations, lower job ratings 
might be a function of inability to work axone or inappropriate attempts 
to establish a dependent relationship with the employer. Clearly, more 
research is needed to determine if this phenomenon indeed exists and why. 

It is hardly surprising that the use of varied instructional techniques 
and outside resources is related to student satisfaction. Students be- 
come bored in a class wherein only lectures are used. Having written 
objectives for the cours**s was relatetl to sL^^dent satisfaction, no doubt 
because students want to know exactly what is expected of them. Finally, 
an interesting finding is that satisfaction is higher in programs that 
use a task-analysis as a basis for instruction. Perhaps this is because 
such programs provide a sequence of training that is accurate and realistic 
with respect to the actual work environment. 
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The positive relationship between the quality of placement services 
and placement rate for females has a simple explanation. That is, jobs 
in technical fields typically biave been more open to men than women. 
The difficulties females experience in obtaining such *obs can be al- 
leviated with an effective placement service. 
Li mitations and S uggesrions for Future Research 

Great care should be taken in interpreting the findings in this 
report. First of all, the speculations presented here are just that — 
speculations. A correlation between a program characteristic and a 
program outcome doesn't even necessarily mean that the characteristic 
ca uses the outcome. Both characteristic and outcome could be a function 
of some third unknown variable. 

Th ere are numerous problems in__reanalyzing archival data, which 
means that one should be careful about accepting the validity of the 
correlations themselves, apart from interpretations. First, there are 
all of the possible errors associated with having a new programmer and 
statistician analyze unfamiliar data. The Department of Vocational and 
Technical Education did provide an interpretive guide, which clearly 
showed which numbers in the data were associated with what variables. 
The problem was in merging the evaluation questionnaire data file with 
the employer rating data file, making sure that the ratings were averaged 
properly and matched with the appropriate p.^ogram data set. Spot checks 
appeared to show that this procedure as accomplished successfully, but 
with such a complicated process, errors are always possible. 

Another problem in dealing with this archival data set was finding 
enough items to validly and reliably represent the co' -^onents of the 
quality indicator dimensions. For two of the proposed dimensions of 
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quality — Work Experience and Evaluation — no items were deemed adequate 
for measuring the dimensions. For some of the dimensions of quality, 
only one or two Items were relevant for some of the subcomponents, and 
some of the subcomponents were not represented at al]. It Is a well-known 
psychometric principle that scales vlth only a few Items tend to be in- 
herently unreliable. But with archival data, one has no choice but use 
what Is available. 

For the scales that contained an adequate number of Items, an es- 
timate of reliability (e.g., Cronbach*s alpha) should have been computed. 
This was not possible, due to limited computer space and funds. One has 
to accept on faith that these scalep are reasonable reliable, based on 
Block's (1978) claim that average ratings become more and more reliable 
wlierT more"rateTS~aTe Tised. Ttre iJroblem here, though, is that In com- 
puting average score? for each program, some programs were represented 
by only a few students, and others, by many. Ideally, each program 
should be represented by the sa'^ number of students, with a sufficient 
number (at least 30 per program) to allow one to make statistical Inferences 
with confidence. 

A final sampling problem concerns race and sex. Originally, the 
effect of these variables on outcomes was to hi examined; however, a 
preliminary analysis of the available data showed that the sample was 
predominantly white and male. Statistical inferences using these 
variables might therefore be misleading; consequently, race and sex 
differences were not examined. The effect of these variables should 
be studied in the future, however. 

The above technicai problems might seem devastating, but the fact 
that some consistent and meaningful relationships between program 

is 
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characteristics and outcomes were found indicates that the findings have 
some validity. If the scoring methods were completely unrel5.able and 
invalid, no meaningful patterns would have emerged from the data, yet 
many patterns were found. These findings should not be regarded as 
definitive, given the technical shortcomings of the study, but they 
can be regarded as real. 

Several suggestions are offered for extending and improving the 
present research study. First, if the quality indicator dimensions 
are to be assessed properly, a new evaluation questionnaire would be 
required. The new questionnaire would contain items designed speci- 
fically to cover the 12 dimensions of Quality, with all of their sub- 
components. Next, it would be desirable if one assessment team rated 
all of Khe programs; this would contribute to the reliability of the 
ratings. Finally, programs should be sampled such that sufficient 
numbers of students of each sex and race are represented, so that 
the impact of these two variables can be assessed. Finally, it would 
be useful to sample programs other than those found in high schools 
and vocational schools, to examine the effects of different delivery 
systems. For a description and comparison of some alternative de- 
livery systems for vocational education, we now turn to the second 
study. 



